This disclosure relates generally to the field of three-dimensional (3D) image reconstruction and, more specifically, to a method, a computer program product and a system for 3D wireframe image reconstruction from video.
3D image reconstruction has a variety of potential civilian and military uses, such as 3D urban modeling for virtual globes and simulations, unmanned vehicle navigation and obstacle avoidance, and intelligent scene tracking and aimpoint maintenance. Many tracking scenarios can be handled by a single-target tracker or a two-dimensional (2D) scene tracker, but more complicated scenarios require a tracker that understands the scene's 3D geometry. Assuming that the scene is mostly rigid, the goal is not to track a particular object, but to track the camera itself. This principle is the basis of matchmoving technology used in cinematography and augmented reality applications to insert virtual objects into real camera footage.
Standard 2D video feature trackers and scene trackers often lose track due to motion parallax effects, such as perspective rotation, deformation, and occlusion. Optical flow-based feature trackers suffer from the aperture problem and have trouble tracking features that rotate or deform. Template matchers must segment foreground objects from background objects that appear to move at a different rate. Occlusions result in lost tracks. Correlation-based scene trackers mistake perspective rotation for drift, even when the camera is already aimed perfectly.
A basic 2D scene tracker uses a correlator to measure image shifts from frame to frame, and a Kalman filter to smooth out the jitter and compensate for the drift. The Kalman filter is an efficient recursive filter that estimates the state of a dynamic system from a series of noisy measurements. As a rule, the 2D scene tracker should not detect any drift when the camera is already aimed perfectly at a single spot. However, a simple 2D correlator can mistake perspective rotation for drift because it fails to recognize the scene's 3D geometry.
Different camera tracking applications have specific advantages and challenges that affect implementation. The tracker system described in the present disclosure is geared toward applications such as airborne navigation, targeting, and rapid 3D urban reconstruction, which requires real-time automated operation from a stand-off distance. This requirement precludes human intervention or batch-processing many frames at a time. One advantage of a targeting or a surveillance system is that the camera is usually calibrated in advance so that its intrinsic parameters are known, and even the extrinsic parameter estimates may be available from an inertial navigation system (INS), so the tracker needs only to correct the pointing angles. At significant stand-off distance, small camera rotation can be approximated as image shift, and videos usually lack sharp persistent features like corners preferred by many trackers.
What is needed is a tracker that can reconstruct the static scene's 3D geometry, given a good estimate of the camera motion.